Click any tag below to further narrow down your results
Links
dbt-doctor scans dbt projects for maintainability issues like missing documentation, weak test coverage, schema drift risks, and DAG problems. It generates a health score (0–100) and can run locally, in pre-commit hooks, or as a GitHub Actions quality gate.
- Detects 122 rules across 9 categories including missing docs/tests, stale models, naming convention violations, and governance gaps
- Integrates into CI/CD with GitHub Actions, outputs sticky PR comments, and can fail builds based on error/warning thresholds
- Configurable via presets (default/strict/enterprise), inline suppressions, and .dbt-doctor config file; respects .gitignore and .sqlfluff rules
A 378-page book teaching practical approaches to building ML models by prioritizing data quality over algorithm complexity, covering data collection, cleaning, labeling, and synthetic data generation with Python. Published February 2024, it emphasizes responsible AI and the role of subject-matter experts in model development.
- Shifts focus from algorithm optimization to data quality as the foundation for robust, fair, and interpretable ML models
- Covers concrete techniques: data imputation, cleaning, labeling, augmentation, and synthetic data generation with scikit-learn code examples
- Introduces "small data" concept and strategies for handling missing data, addressing bias, and building ethical AI systems
A review of "Data-Centric Machine Learning with Python" arguing that ML success depends more on data quality than model sophistication, and explaining why this approach has become standard practice. The reviewer extends this to show how LLMs fit into data-centric pipelines as both label generators and sources of weak supervision.
- The shift to data-centric ML happened years ago through landmark projects like Netflix Prize, ImageNet, and Word2Vec—not a recent invention—driven by the internet's planet-scale data generation.
- Data preparation is the real bottleneck in modern ML: cleaning, imputing missing values, extracting features, removing bias, and iteratively refining labels through human-in-the-loop active learning takes far more effort than training.
- LLMs can be treated as replacements for human subject matter experts in labeling and weak supervision, which means the techniques for managing SME bias and inconsistency now apply directly to LLM outputs.
This article introduces Pointblank, a Python library designed to streamline data validation. It emphasizes user-friendly features, automated validation suggestions, and customizable reports to enhance team communication about data quality issues.
- Pointblank's DraftValidation feature uses AI to scan a dataset and auto-generate suggested validation rules, cutting setup time
- Reports are built for team communication, not just error logs, translating validation results into actionable insights
- It works across Polars, Pandas, and SQL databases, with YAML config support for CI/CD pipelines and shared validation rules
- Supports automated alerting (e.g. Slack notifications) when data quality thresholds for warnings/errors are breached
Data trust is not achieved through a single tool but rather through addressing numerous small failures throughout the data lifecycle. The article illustrates this concept using a real-life incident of clickstream data discrepancies, highlighting the importance of rigorous practices in instrumentation, testing, and pipeline management to prevent data trust erosion.
- A real clickstream data discrepancy incident traced back to small failures across the pipeline, not one big bug, illustrates how data trust actually erodes.
- Preventing this requires rigor at multiple stages simultaneously—instrumentation, testing, and pipeline management—since fixing just one won't restore trust.
- Data trust is framed as "death by a thousand paper cuts": it's lost gradually through many minor issues rather than a single catastrophic failure.
Writing SQL queries is straightforward, but creating a reliable system for running them efficiently is complex and often results in poor data quality and operational inefficiencies. Transitioning from ad-hoc scripts to a structured, spec-driven architecture enhances reproducibility, validation, and observability of SQL jobs, ultimately leading to better management of data and costs.
- Ad-hoc SQL scripts scale poorly because they lack reproducibility, validation, and observability, leading to silent data quality issues and wasted compute costs.
- Moving to a spec-driven architecture (defining queries as versioned, declarative configs rather than one-off scripts) makes jobs testable, auditable, and easier to rerun consistently.
- Centralizing query execution through a structured system enables built-in validation and monitoring, catching errors before they propagate downstream.